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PREFACE 

This  paper  presents  the  author *s  attenpt  to  develop  a  short 
range  forecast  verification  for  llavy  use*  It  is  intended  to  be  a 
simple  practical  system  yet  based  on  sound  meteorological  and  sta- 
tistical principles. 

Undertalcen  as  the  thesis  requirement  for  the  degree  of  Ilaster 
of  Science  in  Aerology,  this  paper  v/as  prepared  at  the  United  States 
ITaval  Postgraduate  School,  llonterey,   California,  during  the  academic 
year  1949-1950. 

The  author  is  deeply  indebted  to  Professor  V/.  D.  Duthie,  for  the 
original  suggestion  of  the  subject  and  for  his  valuable  assistance 
during  the  development.  He  also  vrishes  to  acknovrledge  the  assistance 
rendered  by  Associate  Professor  A.  Boyd  Hevrborn  in  the  preparation  of 
the  verification  tolerance  scales. 
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!•  INTRODUCTION 

The  subject  of  forecast  verification  oontinues  to  be  as  contro- 
versial now  as  it  vms  when  the  pioneer in^  work  in  the  field  -v.-as  done  in 
the  last  century.  The  variety  of  opinion  and  counter  opinion  still  exists ^ 
as  there  are  no  authoritative  criteria  for  settling  an  argument  of  such  a 
subjective  nature*  However,  different  systems  will  be  required  to  fit  the 
diverse  uses  of  forecasts. 

In  this  paper  the  requirenents  of  a  system  vriiich  meets  the  particular 
needs  of  the  United  States  Naval  Aerological  Service  are  set  up.  The  various 
verification  schemes  proposed  and  used  in  the  past  are  examined.  The  simple 
percentage  of  correct  forecasts  is  rejected  for  several  reasons.  The  need 
for  some  logical  basis  of  comparison  is  then  established. 

Three  classes  of  these  bases  for  comparison  are  examined.  They  are 
climatology,  pure  chance  and  allied  computations,  such  as  skill  score,  and 
persistence.  Persistence  is  established  as  the  most  logical  practical  basis 
for  comparison  for  the  Navy  purposes  in  verification  as  set  forth  here.  In 
fact,  the  first  two  schemes  of  comparison,  climatology  and  cheuice,  are 
positively  rejected  as  unsuited  for  either  theoretical  or  practical  reasons 
or  both.  Climatological  evidence  is  presented  in  support  of  the  above 
contention. 

Then  with  the  basis  for  the  verification  system  decided  upon  as  a 
comparison  with  persistence,  the  details  are  developed  in  accordance  with 
the  laws  of  probability  and  statistics. 
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Before  fiLn  objective  verifioation  can  "be  attempted  the  terms  in 
■which  the  forecast  is  to  be  made  must  be  rigidly  defined.  The  exact 
form  of  the  56-hour  forecast  is  therefore  specified.  Some  features  of 
the  present  Navy  forecast  form  vrere  retained  but  many  innovations  are 
made  vdiich  should  increase  the  preciseness  and  completeness  with  vriiich 
the  forecast  is  stated  by  the  forecaster. 

The  development  of  the  tolerance  tables  is  then  presented.  The 
degree  of  correctness  of  the  forecast  is  examined  Ydth  each  meteorologi- 
cal element  considered  separately.  Differences  "vrith  the  present  Ilavy 
system  are  evident  here.  Precipitation,  cloud  cover,  and  visibility 
are  each  considered  separately. 

The  verification  score  is  determined  for  each  of  eight  meteorological 
elements  forecast,  viz.  precipitation,  average  cloud  cover,  lowest  ceiling, 
lovrest  visibility,  surface  iTind  direction,  average  hourly  wind  velocity, 
maximoum  single  gust,  and  maxinium  or  minimum  temperature  as  appropriate. 
The  score  depends  on  two  things,  amount  of  change  since  the  previous  day 
and  the  degree  of  correctness  of  the  forecast.  The  gradations  of  the 
tables  in  degree  of  correctness  are  matched  with  the  limits  of  observerable 
or  predictable  accviracy.  The  logical  basis  for  using  the  amoxmt  of  change 
from  the  weather  of  the  previous  day>  i.e.,  persistance,  as  a  criterion  of 
difficulty  of  a  forecast,  is  established.  The  ntonerical  score  obtained  by 
adding  the  scores  for  each  forecast  element,  read  in  the  appropriate  toler- 
ance table,  has  no  percent  significance.  By  taking  a  score  of  zero  for  a 
correctly  forecast  persisteuit  occurrence,  the  superiority  or  deficiency  of 
the  forecast  compared  with  persistance  is  automatically  established. 


(2) 


The  verification  system  is  then  tested  on  a  series  of  forecasts 
for  typical  IJavy  stations,  illustrating  the  similar  total  scores  ob- 
tained from  the  verification  of  forecasts  ranging  widely  in  difficulty. 

Finally,  the  proposed  forecast  form  and  verification  system  are 
compared  with  the  system  now  in  use  by  the  Naval  Aero  logical  Service* 
The  preciseness  of  the  terms  in  which  the  forecast  must  be  stated  in- 
creases its  value  to  the  consumer  and  the  time  required  for  verification 
compares  favorably  with  the  present  system* 
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II •  RSQUIREIMTOS  OF  A  VERIFICATION  SYSTEIJ  FOR  MVY  USE 

In  order  to  deviso  a  forecast  verification  system  for  Navy  use,  it 
is  first  desirable  to  enumerate  the  uses  of  verification  of  forecasts. 
From  these  uses,  the  particular  qualities  most  desirable  can  be  derived* 

The  principal  uses  of  verification  should  be  in  studies  by  vfhich 
forecasts  could  be  improved.  A  study  of  errors  may  indicate  consistent 
trends  tovrscrd  too  radical  or  too  conservative  forecasts  of  particular 
iweather  elements. 

Another  'iri.despread  use  of  verification  is  in  determining  the  maximum 
period  for  vrtiich  forecasts  are  of  value  by  establishing  a  significant  min- 
imum score.  Also  forecasters  are  frequently  ranked  in  ability  according 
to  their  accximulated  forecasting  average. 

These  uses  in  themselves  may  not  appear  to  justify  the  time  and 
energy  required  in  verification.  It  is  felt  that  accurate  verification 
records  will  provide  incentive  for  individual  forecasters  to  increase 
their  skill  -while  the  absence  of  verification  mil  have  a  contrary  effect. 

These  rather  general  uses  of  verification  require  several  restrictive 
specifications  vjhen  considered  from  a  Navy  viewpoint.  Any  verification 
system  will  inevitably  be  used  to  rank  forecasters  at  one  station  and 
possibly  among  several  different  stations.  The  system  must  be  based  upon 
and  developed  by  sound  statistical  principles.  It  is  desirable  also  that 
the  score  obtained  by  verifying  the  forecast  reflect  only  the  skill  of  the 
forecaster.  This  requires  a  system  that  v/ill  give  comparable  scores  at 
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one  station  for  forecasts  made  in  a  variety  of  synoptic  situations  of 
•widely  varying  difficulty.  In  addition  the  system  should  reflect  only 
forecasting  skill  vrhen   applied  in  climates  ivhere  the  weather  is  very 
changeable  as  vrell  as  those  in  vrhich  interdiurnal  variation  is  slight. 
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Ill,  TYPES  OF  VERIFICATION  SYSTEIS  AJID  THEIR  HISTORICAL  BACKGR0I3ND 

CompletenoBs  and  continuity  in  the  historical  dovelopment  of  veri- 
fication systems  and  their  classification  is  largely  made  possible  by  the 
excellent  survey  of  the  literature  by  Mailer  \_5^»     This  is  especially 
true  of  the  discussion  of  the  contributions  of  European  meteorologists 
prior  to  1^20. 

The  earliest  verification  systems  were  based  on  a  simple  computation 
of  percentage  of  correct  forecasts.  This  system  is  referred  to  as  the 
percentage  system*  A  forecast  was  judged  to  be  a  complete  success  (hit) 
or  a  complete  failure  (miss)  according  to  fixed  tolerances.  For  exan^le, 
with  a  tolerance  of  4  degrees,  a  forecast  of  any  temperature  41  to  4? 
degrees  inclusive  would  be  judged  a  complete  and  equal  success  if  the 
observed  temperature  were  45  degrees*  Any  other  forecast  temperature 
would  be  scored  a  failure*  This  is  the  percentage  system  with  fixed 
tolerances*  This  rather  naive  method  is  still  in  use  by  some  weather 
services  although  the  meaninglessness  of  percentage  of  hits  as  a  test 
of  skill  in  forecasting  was  pointed  out  by  Eoppen  r4j  as  early  as  1906* 

Modern  users  of  this  method  commonly  employ  rating  scales,  assigning 
points  according  to  the  degree  of  success  of  the  forecast*  These  ratings 
are  then  converted  to  percentages.  The  Naval  Aerological  Service  is  one 
of  the  very  few  still  retaining  single  percentage  verification  with  fixed 
tolerances* 

The  second  group  of  verification  s^-^tems  includes  those  in  rrfiich  the 
score  is  determined  by  the  degree  which  the  success  of  sj-noptic  forecasts 
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differs  from  the  success  of  some  type  of  comparison  forecast.  This 
group  is  divided  into  types  accordinr;  to  the  kind  of  comparison  forecast 
used*  These  types  involve,  (a)  elimination  of  pure  chance;  ("b)  comparison 
with  some  kind  of  persistence  forecast;  and  (c)  comparison  vrith  some  kind 
of  climatolosioal  forecast. 

Koppen's  original  suggestion  was  a  comparison  with  random  forecasts. 
This  would  eliminate  the  portion  of  the  forecast's  success  due  to  chance. 
The  current  use  of  skill  score  is  another  example  of  the  elimination  of 
chance  successes.  However,  the  use  of  skill  score  is  limited  to  variates 
capable  of  representation  in  a  tetra-chloric  distribution,  such  as  those 
which  can  be  analyzed  on  a  simple  occurrence  or  non-occurrence  basis. 
This  type  of  analysis  applied  to  thunderstorms  or  precipitation  or  other 
discrete  weather  phenomena  is  very  effective. 

A  second  type  of  comparison  forecast  was  developed  by  Heidke  f  5J» 
He  accepted  Koppen's  view  that  percentage  of  hits  is  an  inadequate  measure 
of  forecasting  skill.  He  proposed  a  system  based  on  using  the  previous 
day's  weather;  i.e.,  persistence,  as  the  comparison  forecast.  The  actual 
verification  score  was  obtained  from  complicated  formulae.  An  interesting 
variation  of  this  method  was  proposed  by  Dinies  F  2 J  for  use  in  the  German 
■"yTeather  Service.  The  observed  weather  of  the  particular  day  in  the  previous 
year  "vms  used  for  comparison. 

More  recently  a  third  type  of  comparison  forecasts  has  come  into  limited 
use.  These  systems  use  climatological  records  as  a  point  of  reference. 
Clayton's  F  1 1  extensive  work  and  the  method  proposed  by  the  Verification 
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Section  of  the  Vfeather  Division,  Headquarters  Army  Air  Force  [vjare 
examples  of  this  use  of  cliinatolop;y« 

In  lfuller*s  entire  survey  of  the  literature  on  verification,  only 
one  author  included  has  published  articles  expressing  the  belief  that 
forecasts  should  not  be  verified,  Schmauss  L^Ji  Although  there  is 
sone  modem  support  for  this  theory,  certainly  the  great  majority  of 
meteorologists  recognize  the  need  for  objective  forecast  verification. 
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lY.  SELECTION  OF  TYPE  OF  VERIFICATION  SYSTEM  BEST  FITTED  TO  NA\T  USES 

The  first  and  simplest  type  of  verification  system  is  based  on  a 
calculation  of  the  peroentare  of  correct  forecasts,  with  each  forecast 
of  an  element  being  considered  as  a  hit  or  a  niss  according  to  a  fixed 
tolerance  table.  This  system  is  presently  used  by  the  Naval  Aerological 
Service  but  it  has  several  serious  faults  when  applied  to  one  station  and 
even  more  when  applied  to  a  variety  of  stations* 

Forecasts  vary  in  difficulty  from  day  to  day  and  from  season  to  season 
at  the  same  station.  Thus  forecast  scores  computed  on  a  basis  of  percentage 
of  hits  will  shovj-  vfide  variation  from  day  to  day,  with  seasonal  trends  prob- 
able, all  independent  of  the  skill  of  the  forecaster. 

The  complexity  and  rapidity  of  vreather  chanres  which, in  f^eneral  in- 
crease vn.th  increasinr^  latitude  and  vary  vrith  {;;eoj;;raphical  location,\7ill 
make  impossible  any  comparison  of  forecasting  scores  from  different 
locations  such  as  the  widespread  Navy  stations. 

The  tolerances  used  to  determine  v/hether  a  forecast  is  a  success  or 
failure  in  the  percentage  system  are  not  suitable  for  stations  with  wide 
geographical  differences.  For  example,  with  a  tolerance  of  4  degrees 
Fahrenheit  in  daily  minimiun  temperature  at  a  maritime  low-latitude  station, 
100  percent  hits  might  be  possible  on  a  repeated  forecast  of  one  particular 
temperature,  while  at  an  inland  high-latitude  station,  great  forecasting 
skill  vrould  be  required  to  obtain  a  score  of  75  percent  hits  on  minimum 
temperature  for  the  same  month. 

Thus  a  system  employing  percentage  of  correct  hits  seems  extremely 
xmlikely  to  fulfill  the  Navy  requirements  as  outlined  in  Chapter  II,  even 
Yri-th  considerable  modification. 
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A  systen  employing  some  type  of  comparison  as  a  reference  point 
and  making  some  allovrance  for  forecast  difficulty  seems  necessary* 
The  elimination  of  tho  successes  achieved  by  pure  chance  is  not  adequate, 
since  it  does  not  change  vfith  changing  forecast  difficulty.  Expectancies 
calculated  on  the  basis  of , climatology  seem  most  logical.  Then  the  correct 
forecast  of  a  rare  event  could  be  given  higher  vroight  than  the  correct  fore- 
cast of  a  common  event* 

An  attempt  -was  made  to  set  up  such  a  system.  Records  were  obtained 
from  several  tj'pical  Kavy  stations  by  recording  tho  various  vreather  elements 
as  they  appeared  on  s^/noptic  charts  for  several  months  of  all  seasons  of  the 
year*  The  variations  in  average  values  and  distribution  of  the  various 
weather  occurences  v/ere  so  great  that  tolerance  scales  for  forecast  score 
computed  on  the  basis  of  long-term  averages  v/ould  have  little  meaning  T;hen 
applied  to  an  individual  season  or  month*  For  example,  advection  fog  at 
Pensacola  had  a  long-term  average  occurrence  of  eight  12-hour  periods* 
The  best  estimate  of  t'.e  expectancy  of  fog  at  Pensacola  for  any  12-hour 
period  -vrould  be  8/56*  Hov/evcr,  if  this  frequency  •were  used  in  scoring 
verification  of  fog  forecasts  in  February  1947  and  February  1^48  it  vrould 
have  little  meaning,  for  in  February  194?  fog  occurred  not  at  all  and  in 
February  I948  it  occurred  during  tv/enty-three  12-hour  periods* 

Temperature  records  are  cited  in  Table  1*  These  data  show  the 
variation  of  monthly  mean  temperatures  about  the  long-term  mean*  This 
tabulation  also  shows  that  the  variation  of  monthly  means  is  so  great  that 
tolerance  intervals  based  on  long-term  means  will  be  too  large  to  be 
practical* 
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UTTER  DIUrciAL  TEIIPERATURE  VARIMIOKS   IN  DEGREES  FAHRENIiSIT 
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B  ITcrr  England 
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Frequency  and  probability  of  occurrence  of  the  various  -woather 
olenents  conputed  after  the  forecast  interval  (month)  is  over  would 
provide  a  lo(;;;ical  basis  for  assessing  difficulty,  Ho^vDver,  this  is 
considered  impractical #  The  forecaster  should  have  prior  knowledge 
of  the  verification  tolerances.  Aerological  office  routine  makes  a 
daily  verification  desirable. 

The  remaining  basis  for  comparison  of  forecasts  is  persistence* 
This  is  less  desirable  theoretically  than  probabilities  computed  post 
facto,  but  is  much  more  practical  in  application*  Day-to-day  persis- 
tence was  chosen  as  being  the  most  probable  as  well  as  most  convenient 
standard*  Persistence  has  the  great  advantage  of  liniversal  application 
to  any  clijnate*  The  fact  that  the  most  probable  vreather  for  tomorrow 
is  that  T/hich  occurs  today  is  supported  by  the  data  of  Table  2* 
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V,  DET^LOPLEl'IT  OF  TOLERAIICE  TABLES  MD  SCORIIIG  SYSTEM 

Having  selected  persistence  as  the  basis  for  comparison,  the  details 
of  the  yerification  system  remain  to  be  decided  upon*  Several  principles 
have  been  selected  as  giuides  in  the  development  of  the  verification  schemes 
for  the  various  -weather  elements  of  the  conpleto  forecast.  These  are  brief- 
ly discussed  here  along  vri-th  the  specific  purposes  they  serve. 

In  the  forecast  and  verification  system  presently  used  by  the  ITaval 
Aerological  Service,  the  method  of  presentation  of  state  of  the  weather 
is  considered  particularly  weak,  A  prescribed  set  of  terms  are  available 
in  this  system  and  one  of  these  must  be  chosen  by  the  forecaster  and 
entered  in  the  appropriate  space  in  tho  forecast  form.  These  terms  range 
from  definite  events,  such  as  thxuider  shower  and  shofrers  to  very  indefinite 
terms,  such  as  "mostly  fair"  and  "tloreatening" •  These  last  two  terms  are 
defined  as  a  certain  range  of  cloud  cover  v;-ith  a  small  amount  of  precipi- 
tation permitted.  Fog  (defined  as  visibility  below  a  certain  minimum  for 
a  certain  duration)  is  another  permissable  term.  Thus  in  this  system, 
precipitation,  cloudiness,  and  visibility  are  all  considered  in  one  fore- 
cast element,  the  state  of  the  weather.  The  result  can  only  be  confusion 
and  \mcertainty  in  the  minds  of  those  who  must  use  the  forecast. 

This  difficulty  will  alv/ays  be  present  when  a  word  or  short  phrase  is 
used  to  summarize  the  weather  conditions  of  a  period  of  time.  A  maxiraian  of 
clarity  can  be  obtained  by  stating  the  forecast  for  each  weather  element 
(precipitation,  cloud  cover,  visibility,  etc.)  separately.  The  short  sum- 
mary spaces  of  the  forecast  form  must  be  filled  v/ith  specified  terms  whose 
meaning  is  clearly  defined  in  tho  minds  of  the  user  as  well  as  the  forecaster. 
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In  the  proparation  of  the  tolerance  alloivonces  of  the  proposed 
system  it  is  assiomed  tlaat  the  most  probable  occurrence  for  any  forecast 
weather  element  for  the  follo-wlng  day  v/ill  be  the  observed  v/cather  of  the 
present  day#  It  is  further  assumed  that  the  distribution  of  tomorrow's 
weather  values  about  today's  is  normal.  This  is  supported  by  the  data  of 
Table  2*  Although  only  temperature  data  are  exhibited,  shorter  tests  were 
r^on  with  other  meteorological  elements  and  the  results  were  cor^arable* 

The  forecast  elements  are  separated  into  three  classes  according  to 
their  type.  Precipitation  is  analyzed  on  an  occurrence  or  non-occurrence 
basis*  iTind  direction,  cloud  cover,  and  temperature  are  continuous  var- 
iates  vrith  no  variation  in  meteorological  iu^ortance.  That  is,  a  North- 
west wind  direction  is  not  considered  any  more  or  less  important  than  any 
other  wind  direction*  The  third  class  of  variates  includes  ceiling,  visi- 
bility, and  T/ind  velocity  which,  although  continuous,  have  varying  impor- 
tance* For  example,  with  low  visibilities  an  error  of  one  mile  is  more 
important  than  the  same  error  with  15  mile  visibility*  The  change  in 
inportance  v/as  considered  from  a  meteorological  viewpoint*  llo  attempt 
xrsLS   made  to  assess  the  operational  importance  of  various  forecasts*  This 
would  vary  widely  according  to  the  type  of  military  operation  for  which  the 
forecast  is  intended*  For  example,  for  high  level  photography  an  overcast 
at  5000  feet  might  be  considered  bad  weather,  while  for  surface  ship 
operations  it  might  be  quite  unimportant* 

In  all  tolerance  tables  an  attempt  was  made  to  verify  to  the  same 
accuracy  as  the  observation  is  made*  Yjhen  the  observed  maximum  temperature 
is  65*^,  a  forecast  of  65^  should  get  more  credit  than  any  other  forecast, 
with  the  score  decreasing  rapidly  with  increasing  error* 
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A  fundeunental  question  remaining  is  the  scoring  system.  7/hat  ratio 
should  be  chosen  "between  the  credit  allo-wed  for  a  correct  persisteiace 
forecast  and  the  credit  allowed  for  a  correct  change  forecast? 

The  persistent  vreather  of  low  latitude  stations  is  reflected  in  the 
high  forecasting  scores  obtained  there  with  fixed-tolerance  percentage 
verification*  The  rapidly  changing  weather  of  higher  latitudes  causes 
lower  scores  Tjhen  forecasts  are  rated  on  the  sane  systen*  This  inequality 
is  independent  of  forecast  skill  and  should  be  removed* 

After  a  study  of  percentage  verification  records  from  high  and  low 
latitude  stations  a  ratio  of  six  to  ten  was  chosen*  This  value  was  computed 
from  the  average  percentage  scores  at  stations  in  both  types  of  geographical 
location* 

The  tolerance  tables  employ  these  ratios*  The  decrease  in  credit  with 
increasing  error  is  computed  from  ordinates  of  the  nonaal  curve*  Using  a 
hypothetical  forecast  element  as  on  example,  the  scoring  for  errors  from 
0-4  imits  from  the  normal  ordinates  would  be: 

Error  in  units         Score 

0  10 

1  9*6 

2  8 
5  5.4 
4                5 

In  order  to  avoid  fractions  the  scores  are  doubled  and  roxmded  off  to 

the  nearest  vrtiole  number  according  to  standard  rounding  procedure*  The 

above  example  then  becomes; 

Error  in  units         Score 

0  20 

1  19 

2  16 

3  11 

4  6 
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The  absciGcao  ii  '^ho  tolerance  table  ar-  ^-'-ted  by  the  inverse 
normal  ordinate  according  to  the  amount  of  chajige  fron  previous  day« 
In  the  hypothetical  example,  the  zero  error  row  mif;ht  read  across: 

Difference  fron  yesterday  in  units 
Error        0   1   2   p   4 
0         12  1?  14  17  20 

Then  the  complete  table  for  this  simplified  illustration  vrould  be: 

Difference  from  yesterday  in  units 
Error 
in  units       0   12^4 

0  12  13  14  17  20 

1  6  7  10  16  19 

2  0  0  5  11  16 
5  0  0  0  7  11 
4  0  0  0  4  6 

The  point  at  vfhich  the  forecast  has  zero  value  is  arbitrarily 
selected  for  each  table.  T\TO--imy  tables  such  as  the  one  above  are  used 
to  verify  wind  direction,  cloud  cover,  and  temperature.  The  score  is 
determined  by  the  error  in  the  forecast  and  the  persistence  of  the  element. 

In  the  cases  of  those  elements  with  varying  natural  importance  a 
three-dimensional  table  is  Tjsed.  This  requires  two  plane  tables  ivhich  de- 
termine forecast  score  from  three  variables:  the  meteorological  importance 
of  the  forecast,  the  amount  of  persistence,  and  the  error  in  the  forecast. 

The  score  for  a  correct  persistence  forecast  is  now  translated  from  a 
value  of  twelve  to  zero.  This  serves  two  purposes.  The  correct  persistence 
forecast  atitomatically  scores  zero,  making  comparison  possible  vd.bhout 
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computation.  Also  it  ronoves  any  possitility  of  the  final  score  being 
confVised  -with  a  percontaj;:e  score  on  the  basis  of  100  percent  for  a 
correct  forecast*  Several  objective  verification  systems  in  the  past, 
notably  those  of  Heidke  and  Claybon,  have  been  considerably  criticized 
by  othor  forecasters  solely  because  the  numerical  score  computed  by  their 
system  vra.s  small  compared  to  100  percent* 

After  this  translation  of  reference  point,  the  example  above  would 
appear  as  follows: 

Difference  from  yesterday 
Error  0    12^4 

0  0    12    5    8 

1  -6-5-2    4    7 

2  -12  -12  -7-14 
5  -12  -12  -12  -5  -1 
4           -12  -12  -12   -8   -6 

Special  details  of  the  scoring  system  as  applied  to  the  individual 
forecast  elements  are  discussed  here. 

!•   Precipitation.  The  type  or  intensity  of  precipitation  is  of  little 
importance  for  llaval  uses  when  the  effect  on  ceiling  and  visibility  is  ade- 
quately forecast.  Quantitative  forecasting  of  precipitation  is  beyond  the 
present  precision  of  the  science*  For  these  reasons  precipitation  forecasts 
are  analyzed  on  a  simple  occurrence  or  non-occurrence  basis. 
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The  only  nodif  icauion  izrcroduced  is  to  cover  cases  in  vriiich  only 
very  small  amounts  of  precipitation  occurred.  In  such  cases  a  forecast 
of  no  precipitation  vrould  be  given  half  credit ♦  J\mounts  up  to  and  in- 
cluding #02  inches  are  defined  as  slio;ht  precipitation  for  this  special 
modif  ication« 

2»   Cloud  Cover •  ITo  special  details. 

5«   Ceiling.  The  meteorologioal  importance  is  assigned  roughly  in  ac- 
cordance \vlth  the  Civil  Aeronautics  authority  requirements  for  instrument 
and  contact  flights.  The  error  is  assessed  in  500  ft.  units. 

4.  "iTind  Direction.  An  eight-point  compass  ivas  considered  adequate  for 
general  meteorological  uses.  Any  forecast  vfith  an  error  of  more  than  two 
points  vra.s  considered  worthless. 

5.  "u'ind  Velocity.  Although  average  velocity  is  computed  to  the  nearest 
knot, .  the  use  of  a  two-knot  interval  in  computing  error  was  necessary  for 
brevity.  The  increased  accuracy  attained  by  the  larger  table  necessary  to 
include  one-knot  intervals  is  not  justified.  The  change  of  scores  Trould  be 
significant  in  only  one  or  t\-ro  places  in  the  entire  table. 

6.  Ilaximim  Gust.  The  present  verification  system  verifies  ma:cim.ura  hourly 
velocity.  This  value  has  little  meaning,  so  maximjom  single  gust  v/as  sub- 
stituted. It  is  forecast  and  verified  to  five-knot  intervals. 

7.  Visibility.  The  meteorological  iraportance  was  again  decided  in  ac- 
cordance with  requirements  for  CAA  Closed,  Instrument,  and  Contact  flight 
regulations. 

8.  I.vaximum  and  lliniraxaa  Temperatvires.  Verification  tables  have  tiTo-degree 
intervals  for  brevity  although  temperature  is  observed  to  the  nearest  whole 
degree.  The  small  errors  occurring  v;here  the  slope  of  the  normal  curve  is 
great  and  the  difference  in  scores  between  spaces  in  the  table  relatively 
large  are  discussed  under  wind  velocity  above. 


VI,  PROPOSED  FORECAST  FORI! 

Before  any  objective  verification  can  be  attempted,  the  forecast 
must  be  stated  in  precise  meaningful  terms.  The  specifications  for  each 
forecast  element  require  a  complete  concise  statement  of  the  expected 
■weather  vrith  no  chance  for  vaipje  terms  or  "hedges".  The  practice  of 
hedging  in  forecasting  is  old  and  vfidespread.  It  is  largely  responsible 
for  the  popular  misxmderstandings  of  the  possible  achievements  and  also 
the  limitations  of  the  science*  In  this  proposed  verification  system  the 
maximira  score  is  attained  only  by  a  very  accurate  forecast, 

vrith  forecast  systems  that  permit  indefinite  complicated  tenuis  such 
as  "mostly  fair",  there  is  a  strong  tendency  for  forecasters  to  attempt  to 
include  in  their  forecast  all  possible  synoptic  developments,  Tjhen  fixed 
tolerance  systems  are  not  clearly  thought  out  there  may  exist  favored 
numerical  forecasts.  For  example,  in  the  system  in  current  llaval  use  the 
tolerances  change  in  too  large  intervals.  Thus  -with  an  observed  average 
■wind  velocity  of  ten  knots  the  allovmnce  for  a  success  is  plus  or  minus 
two  knots  while  the  allowance  for  a  hit  -vvith  an  observed  average  velocity 
of  eleven  knots  jumps  to  four  knots.  This  favors  forecasts  of  nine  or 
eleven  knots  and  makes  the  use  of  ten  knots  penalize  the  forecaster, 
Numerous  other  cases  of  this  inconsistency  exist  in  the  current  system. 

In  this  proposed  forecast  and  verification  system,  the  forecast  is 
required  to  be  stated  in  simple  precise  fashion  and  the  verification 
scoring  does  not  encourage  any  attempt  to  hedge. 
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The  present  box  system  of  NavAer  447(a).  oontaininc  a  brief  statement 
of  the  values  to  be  verified  is  considered  desirable*  These  boxes  contain 
the  forecast  in  explicit  unambiguous  terns,  readily  available  for  verifi- 
cation* These  boxes  are  arranp;ed  in  a  vertical  coluinn  with  the  detailed 
forecast  opposite  them. 

The  forecast  form  suggested  here  is  for  a  12-hour  forecast  interval. 
Three  of  these  fonns  v/ould  ordinarily  comprise  a  normal  forenoon  forecast. 
However,  tvra  forms  for  day  and  following  night  could  be  used  for  a  prelimi- 
nary (early  morning)  forecast  smd  the  length  of  the  forecast  period  could  be 
extended  hy  adding  more  of  the  basic  12 -hour  forms. 

Figure  1  is  an  example  of  the  12-hour  forecast  form.  Figure  2  is  the 
same  form  containing  a  sample  forecast.  The  specifications  for  the  forecast 
are  set  forth  below. 

Precipitation.  Box;  yes  or  no  as  appropriate.  Detail:  if  box  contains 
yes  specify  type  or  t^'pes  of  precipitation,  time  of  beginning  and  ending  if 
within  forecast  period,  intensity  and  changes  in  intensity,  and  amount 
of  precipitation  expected  in  period.  The  special  classification,  light 
precipitation,  is  intended  for  use  only  in  forecast  verification. 

Sky  condition.  Boxes:  state  average  cloud  cover  in  tenths  for  period 
in  upper  box.  In  lower  box  specify  lowest  ceiling  expected  to  occur  for  two 
successive  observations.  Detail:  specify  chronological  variation  of  cloud 
cover.  State  maximum  and  minimum  number  of  tenths  expected  and  give  times 
of  occurrence.  Specify  chronological  variation  of  ceilings,  including  high- 
est and  lovrest  ceilings  expected  and  times  of  occurrence.  Include  statement 
of  turbulence  and  icing  v/hen  applicable. 
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Visibility.  Box:  lovrest  visibility  in  miles  expected  to  occur  for 
two  successive  observations.  Detail:  specify  chronological  variation  of 
visibility. 

Surface  vrind*  Boxes:  top  box  for  prevailing  vjlnd  direction  dxiring 
period.  Middle  box  for  average  hourly  velocity  in  knots.  Lower  box  for 
maximum  single  gust  in  Icnots.  Detail:  specify  all  significant  changes  in 
direction,  velocity  or  gustiness.  Include  highest  and  lo-west  hourly  aver- 
ages expected  with  times  of  occurrence. 

Temperature.  Box:  maxijnum  or  minimum  temperat'jire  in  degrees  Falirenheit 
as  applicable.  Detail:  specify  variation  of  temperature  by  giving  forecast 
temperature  for  4-hour  inteirvals  during  period.  Include  time  of  naximcum  or 
minimxm,  and  where  applicable,  the  time  at  which  the  temperature  is  expected 
to  reach  freezing  point  (32*^)  whether  rising  or  falling. 

Winds  aloft.  Hot  verified.  Forecast  for  tvio   times  to  coincide  with 
local  pilot  balloon  observations  during  forecast  period.  Are  to  be  forecast 
for  6  levels  to  be  chosen  by  the  forecaster  according  to  his  operational 
commitments. 
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Fir.  1  PROPOSED  FORECAST  FORII 


Local  Time  Forecast  Effective 


Precip 
Yes     No 


Precipitation 


Sky  Condition 


Ave.ClouE  Cover 
Tenths 


Lovjest 
Hundreds 


C3 


iling 
of  feet 


Lovrest  Visibility 
Ililes 


Yfind  Dir  ► 


Ave»  ■'.Tin(^ 
ICnots 


U 


IJax.Gust 

ICnots 


Vel. 


Visibility 


Surface  '.Vind 


I;!ax,!an 

Op 


'emperature     Temperature 


ITinds  Aloft 


Time 


Level  1 


Dir-Vel  [Level  4 


Dir-Vel 


Level  1 


Til  ye 


Dir-Vel 


Level  4 


Dir-Vel 


Level  2 


Dir-Vel 


Level  5 


Dir-Vel 


Level  5 


Dir-Vel 


Level  2 


Level  6 


Dir-Vel 


Level  3 


Dir-Vel 


Level  5 


Dir-Vel  iLevel  6 


Dir-Vel 


Dir-Vel  ! 
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2     EXAIvlPLE  OF  TTTffiLVE  HOUR  FORECAST  ON  PROPOSED  FORM 


Forecast  Effective  I80O-O60O  Local 


'Precip 
Yes 


Precipitation 
Light  shovrers  accompanying;;  vreak  cold  frontal  passaj^e  at 
2200  local.  Shovrers  occurring  2000  local  to  2400  local. 
Precipitation  #10  inch  expected* 


Ave .Clou  I  Cover 
8  tenths 


Lovrest  C  3iling 
2000  fee: 


Sky  Condition 
Five  tenths  small  cumulus  clouds  based  at  50OO  ft,  at 
1800  local  increasinj^  to  overcast  vrith  shov-rers.  Ceil- 
ings lowering  to  2000  ft»  in  showers.  Tops  4500  ft.  in-l 


creasing  to  7000 


■i.   o. 


in  showers. Clouds  flattening  and 


decreasing  to  five  tenths  strato-cxamulus  base  2500  fM;» 
tops  5500  ft.  at  0600.  Light  to  moderate  icing  in 
clouds  above  4000  ft.  after  2400.  Light  to  moderate 
turbulence  during  frontal  passage. 


Lowest  visibility  Visibility 

3  Allies  j  Visibility  7  miles  hazy  at  I80O,  decreasing  to 
3  miles  in  shoTrers  2000-2400  and  increasing  to 
15  miles  at  0600. 


7J"ind  Dir; 


Ave  .Ti'ind 
12 


lilJax.Gust 
30 


-i 


Vel. 


Surface  IrTind 
Surface  v/inds  STf  10  -  15  loiots  with  gust  to  20  knots   \ 

veering  to  II^T  with  gusts  to  50  knots  at  2200  local  and  j 

decreasing  slowly  to  IJ\1   8-12  knots  at  060O.  Highest 

hour  15  laiots  2100-2200.  Lov/est  hour  8  knots  O50O-060O. 

liaximum  gust  expected  about  2200  local.  i 


Minimum 
57 


Temperature 


1800  local  49  degrees   2200  local  46  degrees 

0200  local  45  degrees   060O  local  57  degrees  minimum 


030CJ  locaT 


2100  local 


vfinds  aloft 


2000 « 


210-50  - 15000 »   240-45 


5000' 


10000 « 


220-55  i 20000 » 

1 


2000 » 


510-22   ]  20000«   245-40 


250-50 


50001 


255-55 


4- 


50000 ' 


260-65 


10000* 


280-25 


50000*   255-50 


240-50    40000*  j  270-85 
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VII •  DETAIL  OF  VERIFICATION 

!•   Precipitation.  Enter  Table  J>   if  forecast  is  correct  and  record  score* 
2»   Cloud  Cover.    An  average  cloud  cover  in  tenths  is  recorded  for  each 

hour.  A  maxiraum  of  ten  tenths  is  permitted.  The  average 
of  these  values  is  used  for  verification.  Enter  Table  4 
with  the  error  of  the  forecast  in  tenths  and  the  change 
fron  yesterday  in  tenths.  Record  this  score. 
5.   Vfind  Direction.  Record  a  direction  to  the  nearest  of  eight  points  fron 

the  record  of  the  selsyn  recorder.  Use  direction  Tdiich 
prevails  for  greatest  niunber  of  hours.  Enter  Table  5 
with  error  of  forecast  in  points  and  the  difference  in 
direction  from  yesterday  in  points. 

4.  Maximum  or  Ilin- 

imum  Temperature.  Obtain  maximum  or  minijotum  temperature  to  nearest  whole 

degree  from  hourly  and  check  observations  and/or  maximxun 
or  minimum  thermometer  or  thermograph  if  accurate.  Enter 
Table  6  vfith  error  in  degrees  and  difference  fron  yester- 
day in  degrees.  Record  score. 

5.  Average  Hourly 

Wind  Velocity.  From  single  or  multiple  register  record  total  number  of 

knots  passing  anemometer  during  forecast  period  and  obtain 
average  hourly  velocity  to  nearest  whole  number.  Enter 
Table  7  vrlth  this  velocity  and  yesterday* s  average  velocity 
and  obtain  indicator  letter.  Enter  Table  8  vfith  indicator 
letter  and  error  in  !aiots  and  record  score. 
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6#  Ilaxiinum  Gust»    From  Selsyn  Recorder  record  of  "bridled  anemometer  obtain 

maximum  single  gust  during  period*  Enter  Table  7  vrLth 
this  velocity  and  yesterday* s  average  velocity  and  obtain 
indicator  letter*  Enter  Table  8  with  indicator  letter 
and  error  in  laiots  and  record  score* 

7*  Ceiling*        From  ainvays  record,  special,  and  chock  observations 

obtain  lowest  ceiling  occurring  for  tT,TO  consecutive 
obseirvations  ditring  the  forecast  period*  Enter  Table  7 
^irith  this  ceiling  and  yesterday's  lor/est  ceiling  and  ob- 
tain indicator  letter*  Enter  Table  9  with  indicator 
letter  and  error  in  nearest  500-foot  units*  Record  score* 

8*  Visibility*      From  record,  special,  and  check  observations  obtain  lowest 

visibility  occurring  for  two  consecutive  observations  dur- 
ing the  forecast  period*  Enter  Table  7  vrith  this  visibi- 
lity and  yesterday's  minimum  visibility  and  obtain  indi- 
cator letter  and  error  in  miles  and  record  score* 

Add  these  eight  element  scores  to  obtain  final  score* 
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As  an  example  the  sample  forecast  erfiibited  in  Figiire  2  is  verified 
belovr« 

Element      Previous  Ni/^ht    Forecast    Observed    Score 


precipitation 
average  cloud 


none 


yes 


yes 


16 


cover 

5 

tenths 

8 

tenths 

10  tenths 

-  8 

lowest  ceiling 

above  10 j 

,000 

ft. 

2000  ft. 

1200 

ft. 

-  2 

wind  direction 

m 

mi 

SW 

-  8 

ave»  velocity 

7 

12 

14 

-  1 

max.  gust 

12 

30 

28 

1 

min«  temperature 

45 

31 

42 

Total 

-12 

-  4 

It  is  intended  that  the  scores  for  each  12-hour  period  be  kept 
separate  rather  than  be  added  together  or  averaged  in  any  way.  The 
increasing  difficulty  of  periods  farther  away  from  the  forecast  time 
makes  averaging  of  two  or  more  of  these  periods  detract  from  the  mean- 
ing of  the  scores  attained. 
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PRECIPITATION 


Soore 


Correct  Change  Forecast  iG 

Correct  no  Change  Forecast  0 

Incorrect  Change  Forecast 

.Precipitation  less  than  #02  inch           -4 

/Incorrect  no  Change  Forecast 

'Precipitation  less  than  #02  inch          -12 

Other  Incorrect  Forecasts  -24 


TABLE  5, 


CLOUD  COVER 

Change  from  Previous  Day  in  Tenths 

012>45o7   8  9lO 

0002244   68  10  14   16 

1  -2  -2   0   0   2   2   4   6   8  12   14 

Error  in   2  -12  -10-10  -8  -8-6-4   0   4  8   10 
Tenths 

5  -18  -18  -16  -16  -16  -14  -12  -8-4  0    2 

4  -24  -24  -24  -24  -24  -I8  -I6  -14  -10  -6   -4 

5  -24  -24  -24  -24  -24  -24  -24  -I8  -I6  -12  -10 

6  -24  -24  -24  -24  -24  -24  -24  -24  -24  -I8  -I6 

TABLE  4, 
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TEMPERATURE 

Change  from  Previous  Day  in  Degreos  F» 

0-1      2-5      4-5       6-7       8-9  10-11  12-15  14-15  16-17  18 

0-10112254578 

2-5        -6        -5         -5        -2-11  5 

Error  in     4-5       -12       -l2         -8         -7         -6     -5         -2 
Degrees 

6-7       -12       -12       -12       -12       -12     -8         -7 

8-9       -12       -12       -12       -12       -12     -12     -12 

TABLE  5. 


Vraro  DIRECTION 

Change  from  Previous  Day  in  Points 

01254 

001258 
Error  in 
Points  -1  -4  -5-2   1   5 

2  -8  -8  -7  -5  -2 
TABLE  6. 


4 

6 

7 

-1 

1 

4 

-5 

-4 

-1 

0 
-0 

-7 

-6 
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DIDICATOR  LETTERS 


Visibility 
in  Miles 

Gust  in 
Knots 

Average 
Yelocit 
Knots 

1 

0-1 

60 

45 

2 

2-3 

56-60 

26-45 

5 

4-10 

26-55 

16-25 

4 

10 

0-25 

0-15 

dreds  of  feet 

0-5 
Class    2     2-5       56-60      26-45        6-15 

16-50 
50 


Class  Difference  from  Previous  Day 


0 

1 

2 

5 

1 

D 

C 

B 

A 

Class 
Today 

2 
3 

G 
J 

F 
I 

E 
H 

4 

N 

H 
1 

L 

K 

TABLE  7, 
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VERIFICATIOII  SCOP.niG  FOR  VHITD  VELOCITY 


Llaxiraum 

Gust 

Averaf^ 

e  Volooity 

Error 

'  in 

Knot 

s 

Error  in  ICnots 

0 

5 

10 

15 

20 

0-1 

2-5 

4-5 

6-7 

0-9 

10-11 

12-15 

14-15 

16-17 

A     8 

5 

0 

-4 

-7 

8 

8 

7 

6 

4 

0 

-4 

-7 

-10 

3     6 

4 

-1 

-5 

-8 

6 

6 

5 

4 

2 

-1 

-5 

-8 

-10 

C     4 

2 

-1 

-5 

-8 

4 

4 

5 

2 

0 

-2 

-6 

-8 

-10 

D     5 

1 

-2 

-6 

-8 

5 

2 

1 

-1 

-5 

-5 

-8 

-10 

-12 

E     5 

2 

^1 

-7 

-9 

5 

4 

2 

«T_ 

-4 

-6 

-9 

-12 

-12 

F     3 

1 

-5 

-8 

-12 

5 

2 

1 

-2 

-5 

-7 

-9 

-12 

-12 

G     2 

0 

-4 

-8 

-12 

2 

0 

-5 

-7 

-? 

-12 

-12 

-12 

-12 

H     4 

0 

-6  - 

-12 

-12 

4 

5 

-2 

-6 

-9 

-12 

-12 

-12 

-12 

I     1 

-2 

-7 

-12 

-12 

1 

0 

-4 

-7 

-9 

-12 

-12 

-12 

-12 

J     1 

-2 

-7 

-12 

-12 

1 

0 

-4 

-7 

-9 

-12 

-12 

-12 

-12 

K    5 

-4 

-12 

-12 

-12 

5 

1 

-5 

-8 

-12 

-12 

-12 

-12 

-12 

L     1 

-5 

-12 

-12 

-12 

1 

0 

-4 

-8 

-12 

-12 

-12 

-12 

-12 

M     0 

-6 

-12 

-12 

-12 

0 

-1 

-5 

-9 

-12 

-12 

-12 

-12 

-12 

11     0 

-7 

-12 

-12 

-12 

0 

-1 

-6 

-9 

-12 

-12 

-12 

-12 

-12 

TABLE  8, 


(30) 


Lovrest  Ceilin?! 


0 

A 

8 

B 

6 

C 

4 

D 

3 

E 

5 

F 

3 

G 

2 

H 

4 

I 

1 

J 

1 

K 

5 

L 

1 

M 

0 

H 

0 

Error  in  500  feet  Units 

12       3       4  5  6 

5  0-6     -12  -12  -12 

4  _i     -7     -12  -12  -12 

2  -5  -7  -12  -12  -12 
1  -3     -8     -12  -12  -12 

3  -2     -7     -9  -12  -12 

1  -5  -8  -10  -12  -12 
0  -4  -8  -10  -12  -12 
3  0  -3-7  -5  -12 
0  -2  -4  -7  -9  -12 
0  -2     -5     -8  -10  -12 

2  1-2-5  -7  -9 
0  -1    -3    -6  -8  -9 

0    -1  -2    -4    -6  -8  -10 

0    -1  -2    -4    -6  -8  -10 


CEILniG  Mm  VISIBILITY 

Lowest  Visibility 
Error  in  Miles 


01234  5 

8  6-1-7  -12  -12 

(y  5-2-8  -12  -12 

4  3-3-8  -12  -12 
3  2-4-8  -12  -12 

5  3-1-5  -9  -12 
3  2-2-6  -9  -12 

2  0-3-7  -9  -12 

430-3-7-9 

1  0-2-4  -7  -9 

1  0-2-5  -8  -10 

3  2-1-4  -7  -9 
1  0-2-4  -7  -9 
0  -1    -3     -5  -8  -10 
0  -1     -3     -6  -8  -10 


TABLE   9. 
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VIII  •  COIICLUSION 

As  7/as  stated  earlier  the  final  score  computed  as  above  has  no 
percentage  connotation*  The  highest  score  attainable  is   8o.  From 
the  amoxant  of  change  from  previous  day  reouired  it  is  very  unlikely  that 
a  score  higher  than   40  would  be  possible*  If  the  vroather  were  exactly 
the  same  as  the  previous  day  and  -were  correctly  predicted  the  computed 
score  would  be  zero» 

Several  v/riters  have  expressed  the  belief  that  blind  persistence, 
forecasting  no  oliange  day  after  day,  vrould  give  about  50  percent  successes 
scored  by  fixed  tolerance  verification*  This  type  of  pure  persistence  vra,s 
tested  Tfith  the  scoring  proposed  here  and  the  average  score  T/as  -55  • 

In  another  test  a  small  group  of  forecasts  vrere  prepared  and  verified* 
for  a  selected  group  of  naval  stations  involving  a  Tride  variety  of  geo- 
graphical locations*  In  this  test  five  forecasts  were  made  for  the  follow- 
ing daytime  period  O60O-I8OO  and  verified  with  the  proposed  tolerance  tables* 
The  average  scores  Yrere:  Boston  -40,  Pensacola  -55 »  Coco  Solo  -42,  Plonolulu 
-42,  San  Francisco  -59»  Kodiak  ->6*  The  sample  is  certainly  small  and  the 
scores  may  well  reflect  the  personal  forecasting  experience  of  the  author, 
but  it  is  believed  that  they  are  fairly  representative*  The  average  score 
on  these  forecasts  was  -59 •  The  uniformity  of  scores  from  high  and  low 
latitude  indicates  that  the  system  has  compensated  fairly  well  for  varying 
difficulty  of  forecasts*  If  any  bias  still  exists,  it  is  probably  in  favor 
of  the  higiier  latitude  stations  with  their  more  difficult,  changing  weather* 

The  final  deterriiiiation  of  the  average  attainable  score  and  hoT/  well  the 
system  fulfills  its  primary  purpose  of  measuring  skill  in  forecasting  will 
await  wider  use  and  tests  in  a  much  larger  number  of  cases* 
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The  proposed  forecast  form  is  much  better  suited  to  IJaval  uses  than 
the  present  form  because  of  the  separation  of  forecast  elements  and  the 
completeness  vri.th  vriiich  they  must  be  forecast  in  xiiiequivooal  terms.  This 
forecast  form  v/ill  require  that  the  meteorolo£;ist  have  a  clear  picture  of 
the  expected  -weather  in  mind  vriien  malcinp;  the  forecast,  as  the  specifications 
call  for  the  values  of  most  vreather  elements  to  be  reduced  to  numbers  stating 
highest,  lov/est,  and  average  valines  expected  and  to  state  as  oloGely  as  pos- 
sible the  time  of  occurrence*  Definiteness  as  to  event  and  time  of  occurrence 
diminish  as  the  period  of  time  of  the  forecast  increases.  However,  a  fore- 
caster should  be  able  to  specify  the  weather  vfithin  close  limits  for  at  least 
three  of  the  proposed  12-hour  periods. 

Another  use  of  this  verification  system  is  in  the  evaluation  of  the 
so-called  objective  or  "mechanical"  forecasts.  As  long  as  these  forecasts 
are  used  in  definite  forecasts  of  an  occurrence  or  non-occurrence  type  there 
■would  be  no  significsJtit  advantage  in  using  this  type  of  verification.  The 
type  of  forecasts  made  according  to  inflexible  procedures  and  from  predictants 
whose  connection  with  the  variable  forecast  is  not  knovm  should  be  confined  to 
forecasts  of  a  ims   -  no  type. 

Y/hen  mechanical  techniques  are  used  to  predict  the  value  of  continuous 
variates,  such  as  wind,  sVcy     condition  or  temperature,  any  defects  will  be 
obvious  v;hen  the  forecasts  are  verified  by  this  proposed  system.  The  lovf 
scores  occurring  as  a  result  of  the  large  errors  made  oy  a  mechanical  S2,^ten 
will  overbalance  their  sviccesses,  especially  if  the  system  does  not  contain 
all  significant  predictants  vrhich  affect  the  weather  occurring. 
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The  chief  advantage  of  the  verification  systen  over  the  one  presently- 
used  is  that  the  final  score  more  nearly  reflects  only  forecasting  skill. 
The  scores  obtained  on  successive  days  at  the  sarae  stations  and  those  ob- 
tained at  different  stations  are  comparable.  The  verification  systen  offers 
incentive  to  the  forecaster  at  both  tropical  and  hi{;;h  latitude  stations,  re- 
quiring definite  effort  at  both  places  to  obtain  high  scores.  The  verifi- 
cation system  leads  the  forecaster  to  state  the  forecast  as  precisely  as 
possible,  where  the  present  system  encourages  hedging  vfith  indefinite  terms 
and  uses  tolerance  tables  v/hich  favor  certain  numerical  forecasts. 

After  more  use  or  tests  of  the  proposed  verification  system,  it  may  be 
desirable  to  alter  the  degree  of  difficulty  in  some  of  the  tolerance  tables 
by  increasing  or  decreasing  certain  values.  Further  tests  may  shovr  the  de- 
sirability of  a  further  shift  of  the  zero  point  from  the  value  used  here. 
A  possible  new  reference  point  is  the  average  score  attained  by  persistence 
forecasts  instead  of  the  score  of  a  correct  persistence  forecast. 
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